fix(rag): stop reasoning-token starvation that discarded answers as "unsupported"#580
Merged
Merged
Conversation
…unsupported" gpt-5.5 draws reasoning tokens from the SAME max_output_tokens budget as the visible answer. The strong route ran high reasoning effort against a 4000-token cap, so reasoning exhausted the budget and returned `incomplete: max_output_tokens` before writing the answer — the answer was then discarded and the request degraded to "unsupported" after 60-90s (confirmed via live rag_queries telemetry). The same high effort also overran the 30s answer timeout (provider_timeout). It is a class of bug: every reasoning-model call sized max_output_tokens without reserving reasoning headroom, including the ingestion enrichment/extraction paths that ran silently over the corpus. - env: OPENAI_MAX_OUTPUT_TOKENS 4000->16000; OPENAI_STRONG_REASONING_EFFORT high->medium. medium resolves every query class, including the safety-critical medication_dose_risk and table_threshold that strongReasoningEffortForQueryClass kept at the full configured effort and that starved first. - openai: reasoningHeadroomFloor(effort) floors max_output_tokens by effort in responseBody so no call site can under-provision reasoning headroom. Math.max only ever raises a budget, and the budget is a ceiling (billed per token used), so it is free. - rag: truncation self-heal — strong escalations retry at a boosted cap (2x, >=24000) instead of re-truncating on the same budget; a cumulative generation wall-clock budget skips the third (polish) generation to bound tail latency, keeping the valid strong answer rather than risking a truncation -> unsupported tail. - ingestion: document-enrichment and model-index-extraction now observe truncation and warn loudly (with document identity) instead of silently cutting off enrichment. - observability: answer-slo adds truncation/timeout fallback counters to the /api/health deep probe so this regression class is visible without hand-running SQL. Verified: verify:cheap (1673 tests) + typecheck + prettier offline; live golden retrieval 36/36 (content_mrr@10 0.9414); live eval:quality --rag-only eliminated the max_output_tokens and provider_timeout fallbacks. An A/B baseline (pre-fix vs fix on the same live corpus) is byte-identical on every grounding/citation/failure metric (zero quality regression) and cuts p95 generation latency ~14s (75.9s -> 61.5s). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
BigSimmo
enabled auto-merge (squash)
July 13, 2026 12:22
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (10)
📝 WalkthroughSummary by CodeRabbit
WalkthroughGeneration now preserves structured-result metadata, warns on truncation, reserves reasoning headroom, expands strong retry budgets, limits cumulative retry time, and tracks truncation and timeout fallbacks in answer SLO metrics. ChangesGeneration resilience
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant generateWithModel
participant OpenAIResponsesAPI
participant strongQualityRepair
generateWithModel->>OpenAIResponsesAPI: Request structured output
OpenAIResponsesAPI-->>generateWithModel: Return text and truncation status
generateWithModel->>strongQualityRepair: Evaluate answer quality
strongQualityRepair->>OpenAIResponsesAPI: Retry with larger output budget
OpenAIResponsesAPI-->>strongQualityRepair: Return retry result
Possibly related PRs
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
✨ Simplify code
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5874814cd3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| : null; | ||
| const answerNeedsStrongQualityRepair = usedStrongModel && Boolean(strongQualityFailureReason); | ||
| if (answerNeedsStrongQualityRepair) { | ||
| if (answerNeedsStrongQualityRepair && generationLatencyMs >= generationTotalBudgetMs) { |
There was a problem hiding this comment.
When the fast/strong path has already spent just under 2 * OPENAI_ANSWER_TIMEOUT_MS, this >= check still allows the strong_quality_retry below to start with a full per-call timeout, so two 29s generations under the default 30s timeout can still kick off a third 30s call and recreate the ~90s tail this guard is meant to prevent. Reserve the next call's timeout in this guard, or pass only the remaining budget into the retry, so the new wall-clock budget actually bounds the chain.
Useful? React with 👍 / 👎.
This was referenced Jul 13, 2026
BigSimmo
added a commit
that referenced
this pull request
Jul 13, 2026
… baseline (#622) * fix(eval): recalibrate canary latency budgets to the post-#606 honest baseline The pre-2026-07-13 budgets (p95 25s overall, 12s extractive) were calibrated while confidence_gate_blocked refusals dominated the answer subset - refusals return in ~1-2s, so the canary was timing fast wrong answers. With #606 restoring grounded answering, p95 reflects real work: the fast->extractive fallback chain spends the full 30s answer timeout before stitching sources, and #580's strong truncation self-heal can run two sequential generations. Post-fix observed subset p95 is 48.3s with every quality metric green (grounded 1.0, citation failure 0, numeric failure 0). New budgets catch regressions from that baseline instead of re-flagging the fix: overall 60s, extractive 50s, strong 60s. Fast (25s) and unsupported (4s) are unchanged - refusals must stay fast, which is exactly the waste mode #580 eliminated. These are eval-runner budgets (GitHub-hosted, cross-region to Sydney), not production UX targets; production SLOs stay in answer-slo.ts and docs/observability-slos.md. Closes the remaining red on #459. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(eval): bucket generation-fallback chains separately; sync reindex gate ceiling Review follow-ups on the latency recalibration: - latencyRouteForAnswer now classifies answers whose routing reason records a generation_fallback into a dedicated "fallback" latency bucket (50s budget): a failed generation structurally costs its timeout plus the fallback work. This restores the tight budgets the first commit accidentally loosened or missed: extractive returns to 12s (no-model stitching must stay fast) and plain fast stays at 25s, so regressions on either path cannot hide inside the fallback allowance. - reindex-eval-gate quality p95 ceiling 25s -> 60s to match the canary budget (a no-regression reindex at the honest ~48s baseline must not be blocked by the absolute bar); baseline-relative regression detection via latencyRegressionMs is unchanged. Runbook updated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(eval): classify generation fallbacks before strong-route reasons A strong-classified reason carrying a generation_fallback (e.g. multi_document_comparison_synthesis; generation_fallback:provider_timeout) must budget as a fallback chain, not blend into the strong budget; successful strong generations (including quality retries) never record a generation_fallback, so genuine strong answers are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Production answers were degrading to "unsupported" after 60–90s on high-value clinical queries. Root cause (confirmed via live
rag_queriestelemetry): gpt-5.5 draws reasoning tokens from the samemax_output_tokensbudget as the visible answer. The strong route ran high reasoning effort against a 4000-token cap, so reasoning exhausted the budget and returnedincomplete: max_output_tokensbefore writing the answer — the answer was discarded. The same high effort also overran the 30s timeout (provider_timeout). It's a class of bug: every reasoning-model call sized its token budget without reserving reasoning headroom, including the ingestion enrichment/extraction paths that ran silently over the whole corpus.OPENAI_MAX_OUTPUT_TOKENS4000→16000;OPENAI_STRONG_REASONING_EFFORThigh→medium. medium resolves every query class — including the safety-criticalmedication_dose_risk/table_threshold, whichstrongReasoningEffortForQueryClasskept at the full configured effort and which starved first.reasoningHeadroomFloor(effort)floorsmax_output_tokensby effort inresponseBody, so no call site can under-provision reasoning headroom.Math.maxonly ever raises a budget, and the budget is a ceiling (billed per token used) — free unless hit. One change fixes every call site.document-enrichment+model-index-extractionnow observe truncation and warn loudly (with document identity) instead of silently cutting off enrichment.answer-sloadds truncation/timeout fallback counters to the/api/healthdeep probe.Host/region change (the separate cross-region retrieval latency) is intentionally out of scope.
Verification
npm run verify:cheap— green (1673 tests, 1 skipped, 179 files) +typecheck+prettierclean. (Fullverify:pr-localbuild/bundle scan left to CI.)npm run eval:retrieval:quality— 36/36,document_recall@5=1.0,content_recall@5=1.0,content_mrr@10=0.9414(≥ 0.9255 baseline). Retrieval untouched (change is generation-only).npm run eval:quality -- --rag-only(live) — themax_output_tokensandprovider_timeoutfallbacks are eliminated. Ran an A/B baseline (pre-fix vs fix on the same live corpus): every grounding/citation/failure metric is byte-identical (grounded 0.4667=0.4667, citation_failure 0.2955=0.2955,failure_category_countsidentical, zero grounded cases lost) → zero quality regression, plus a ~14s p95 latency win (75.9s→61.5s) and one timeout fallback removed. The grounded-rate threshold miss is proven pre-existing (documentation-lookup confidence-gate cases, e.g.discharge-documentation).npm run check:production-readiness— not run (provider-backed / confirmation-gated per repo policy). No clinical-decision, privacy, Supabase, or source-governance behavior changed by this PR.Clinical Governance Preflight
Clinical KB Database(sjrfecxgysukkwxsowpy) — untouchedNotes
main(onlytests/answer-slo.test.tsneeded a trivial merge with main's newevent_typescoping — the truncation/timeout counters inherit that filter via the sharedbase()).🤖 Generated with Claude Code